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ABSTRACT 



A study at the United States Air Force Academy examined 
reasons for "survey saturation" among students. Despite relatively high 
return rates, anecdotal student data suggested that students felt burdened by 
frequent administration of surveys; and administrators expressed concerns 
about the general validity of survey data. Information was gathered on 
students' views of an optimal survey, their concerns about confidentiality 
and the use of electronic surveys, and student motivation and survey 
participation. Written comments gave insights into issues emphasized by 
students and provided evidence that survey responses impact upon policy 
decisions. The survey data provided clear empirical support for students' 
feelings of being over- surveyed and the resulting negative impact on survey 
validity. Findings from the report suggest consideration of important 
guidelines for future survey design and administration. (Contains 8 
references.) (CH) 



******************************************************************************** 



Reproductions supplied by EDRS are the best that can be made 
from the original document . 



******************************************************************************** 




ED 422 805 



Undergraduate Perceptions 1 



Running head: UNDERGRADUATE PERCEPTIONS OF SURVEY PARTICIPATION 



Undergraduate Perceptions of Survey Participation: 
Improving Response Rates and Validity 1 



Bernhard W. Asiu, Lt Col, USAF 
Director, Institutional Assessment 
United States Air Force Academy 
HQ USAFA/XPR 
2304 Cadet Drive, Suite 300 
USAF Academy, CO 80840-5002 
(719) 333-2587 
asiub.xp@usafa.af.mil 



Miriam L. Fultz, Ph.D. 
Director, Assessment and 
Instructional Research 
University of Southern Colorado 
2200 Bonforte Boulevard 
Pueblo, CO 81001-4901 
(719) 549-2982 
fultz@uscolo.edu 



Christopher M. Antons 

Director, Institutional Research and Assessment 
Mount Saint Mary’s College 
12001 Chalon Road 
Los Angeles, CA 90049 
(310) 954-4226 
cantons@msmc.la.edu 



N 

V) 






PERMISSION TO REPRODUCE AND 
niSSFMlNATE THIS MATERIAL HAS 
BEEN GRANTED BY 

AIR 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 

1 



U.S. DEPARTMENT OF EDUCATION 
Office of Educational Research and Improvement 
EDUCATIONAI RFSOIJRCFS INFORMATION 
. CENTER (ERIC) 

& This document has been reproduced as 
received from the person or organization 
originating it. 

□ Minor changes have been made to 
improve reproduction quality. 



Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 




'Presented to the Association for Institutional Research Annual Forum, Minneapolis, May 1998 




2 



A R 

for Management Research, Policy Analysis, and Planning 



This paper was presented at the Thirty-Eighth Annual Forum 
of the Association for Institutional Research held in 
Minneapolis, Minnesota, May 17-20, 1998. 

This paper was reviewed by the AIR Forum Publications 
Committee and was judged to be of high quality and of 
interest to others concerned with the research of higher 
education. It has therefore been selected to be included 
in the ERIC Collection of AIR Forum Papers. 



Dolores Vura 
Editor 

AIR Forum Publications 




3 



Undergraduate Perceptions 



2 



Abstract 

The institution supports a decentralized assessment program. Data are typically collected via 
attitude and opinion surveys. Despite relatively high return rates, anecdotal student data 
suggested that students felt burdened by frequent survey administrations and did not always 
perceive that survey participation was voluntary (even though a voluntary participation statement 
appears at the beginning of USAFA surveys). Further, administrators expressed concerns about 
these perceptions as well as the general validity of USAFA student survey data. The objective of 
the current research was to gain a thorough understanding of this apparent “survey saturation.” 
Desired survey completion times, annual administration frequency, and survey length suggestions 
were obtained. Written comments provided insight into issues emphasized by students and 
evidence that survey responses have impacted policy decisions. Respondents also suggested 
remedies that can be used to address student concerns regarding the confidentiality of electronic 
survey responses. Suggested guidelines for improving response rates and data validity are 
described and future research areas are discussed. 
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Undergraduate Perceptions of 

Survey Participation: Improving Response Rates and Validity 
The current public, accreditation, and legislative mandates for accountability and 
effectiveness have profoundly impacted higher education (Cameron & Smart, 1998; Darling- 
Hammond, 1994). Like many post-secondary institutions, the United States Air Force Academy 
(USAFA) has become increasingly active in the assessment of educational, institutional, student 
support, athletic, and military training outcomes. Not surprisingly, a significant increase in the 
number of data collection initiatives has been observed recently, usually in the form of student 
attitude and opinion surveys. Hence, herein lies the dilemma: leaders need survey data for 
decision-making purposes, but USAFA students (cadets) rarely agree with or understand the 
relevance of requests for such data. 

Student response to USAFA’s burgeoning survey effort can be characterized as mixed as 
can the students’ motivation toward survey participation. In one sense, the Academy’s military 
training environment tends to foster participation based upon a “sense of duty.” Conversely, this 
same environment may not ensure sincere or “accurate” student responses for a variety of 
reasons, for example, respondents do not always receive timely survey feedback regarding the 
impact of their participation. From a perspective external to the cadets, recommended survey 
research practices (e.g., advance notification, observable management support) have been 



consistently implemented by the Academy’s Survey Program (Cook & Campbell, 
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Kosecoff, 1985; Fowler, 1995; Oppenheim, 1992; Robinson, Shaver, & Wrightsman, 1991; 
Rossi, Wright, & Anderson, 1983). 2 



2 Among others, other practices include: providing advanced notice/publicity to respondents and their 
advisors, reserving survey participation time so do not expend their personal time, ensuring that senior leadership 
provides evidence of their sponsorship, sending follow up emails, and encouraging feedback to survey respondents. 
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Over time, informal discussions with students and recent graduates, focus group 
feedback, and responses to open-ended survey items suggested that USAFA faced a challenge. 
Despite the enviable 60% - 80% response rates for the major institutional surveys, this anecdotal 
data highlighted factors that could potentially impact the validity of the survey data: student 
resentment toward questions focused on sensitive issues, distrust regarding assurances of 
response confidentiality or anonymity, resentment toward the time required by survey 
participation, and a lack of evidence about how student input had been used to improve student 
life. Furthermore, this informal data highlighted practices that could be employed to positively 
impact the intrinsic willingness of students toward survey participation. 3 

Generally, the issue can be conceptualized as the students feeling “over-surveyed” or 
burdened by frequent requests for survey participation. To better understand and refine the scope 
of this apparent problem, data were collected from those most impacted by the proliferation of 
USAFA surveys: the students themselves. Ironically, the data collection method was a survey 
about surveys. The primary research questions were: 

(a) Do the students (cadets) feel “over-surveyed?” 

(b) How do students define the term “over-surveyed?” 

(c) What changes could be made to the current survey design and administration 
approach that might result in reduced response burden and higher response rates? 

v.dy nOw do students view response confidentiality/anonymity when electronic survey 
forms are used? 



3 These practices include: developing more comprehensive efforts toward relating survey topics to the 
participants’ daily lives, thoroughly explaining the survey’s purpose and use of the data, communicating program and 
policy changes based upon survey data in a timely manner, minimizing survey complexity and length, and 
emphasizing the confidentiality or anonymity of survey responses. 
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Method 

Instrument Development 

Two student focus groups were conducted to clarify issues and to guide survey design 
efforts. Students were asked to define the term “over-surveyed” and to identify factors that 
contributed to their survey participation decision. Participants indicated their preferences 
regarding the optimal number of questions and time required to complete a survey. Moreover, 
the participants discussed their confidentiality/anonymity concerns in relation to electronic 
survey forms and suggested ways in which assurances of confidentiality would be most 
effectively implemented. 

The final instrument consisted of 10 forced- and one multiple-choice items. 4 In addition, 
eight open-ended items asked students to describe factors that contributed to feeling “over- 
surveyed” and to further describe their preferences in terms of survey length and the time 
required to complete surveys. Other topics included the efficacy of offering rewards for 
participation and enhancing student confidence in the overall confidentiality of the data gathered 
via electronic means. Finally, a list of surveys administered during the previous year was 
developed. For each survey, respondents were asked to indicate whether they had completed the 
survey and to rate their perceived relevance and importance. Survey results for this list relate to 
specific Air Force Academy assessment activities and initiatives. Hence, the results are not 
included or discussed given the purpose of the current report. 



4 The forced-choice items required students to: indicate whether they felt over-surveyed; describe their view 
of an optimal survey (number of items and time required to complete the survey); specify the minimum of surveys 
that should be administered during an academic year; identify their preferred survey completion time (officially 
scheduled time versus personal time; describe their concerns regarding the confidentiality of electronic surveys; 
indicate their willingness to serve as a reviewer for upcoming student surveys. The multiple-choice question asked 
respondents to indicate their reasons for participating or not participating in USAFA survey efforts. 
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Participants 

Six student squadrons were randomly selected from the total population of 40 squadrons. 

Approximately 100 students (or cadets) comprise a squadron. All students are randomly 
assigned to squadrons and all squadrons are housed in a dormitory setting. In essence, the 
squadron serves as the focus of student life. Previous USAFA analyses have consistently 
demonstrated the absence of differences in terms of aptitude and demographic factors such as 
gender and ethnic status. Hence, the squadron served as the unit of analysis during the process of 
randomly selecting the six squadrons. 

Procedure 

Students from the six squadrons completed the survey in their dormitory rooms via a 
local area network. A single 50-minute class period was specifically reserved for this major 
survey administration. The remaining 34 squadrons were randomly assigned to complete one of 
three paper or electronic surveys (which were not related to the current study). This 
administration approach was aimed toward minimizing the response burden, i.e., each student 
completes only one survey instead of four. Previous USAFA experience has shown that 
administration efficiency is achieved and response rates are enhanced by administering four 
surveys during a reserved time slot. 

Prior to the survey administration period, all students received a “motivational” e-mail 
message from USAFA’ s senior leadership asking for their support. All messages were tailored to 
the specific survey to be completed by a given squadron. Generally, each message included a 
description of the survey objectives and how the resulting data would be used. Further, 
respondent were told when and how they could access aggregate survey feedback. This message 
stressed the voluntary nature of survey participation and described the measures enacted to 
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ensure response confidentiality. Respondents were told that the RAOSOFT® program used to 
create the electronic survey automatically encrypted their data. Furthermore, the e-mail 
emphasized the “Academy’s trusted agent” function played by the USAFA Office of Institutional 
Research and Assessment and the security procedures that were in place to enhance the 
confidentiality of survey responses. 

Results 

Demographics 

Of 590 potential participants, 360 students voluntarily completed the survey for a 61% 
response rate. Of the 360, the proportion of respondents for each class year (freshman, 
sophomore, junior, senior) was 30.3%, 27.7%, 23.3%, and 18.7% respectively. The gender of the 
respondents (17% female and 83% male) mirrored the proportions found in the squadrons and in 
the overall student enrollment. 

Do Students Feel “Oversurveved?” 

Ninety-seven percent of the respondents reported feeling at least somewhat “over- 
surveyed.” Nearly half (48%) chose “Yes, definitely.” Of those, sophomores endorsed this 
response most often (62%) while freshmen were least likely to report feeling over-surveyed (8%). 
Twenty-five percent of the respondents chose the more moderate option “Yes, I feel over- 
surveyed, but I realize that the survey data are valuable.” Seniors were most likely to choose this 
response (3 /%>). uther responses included “Somewhat” (13.5%) and “Somewhat, but I realize 
survey data are valuable” (10.7%). 

Student Perspectives: What “Oversurveving” Means 

Respondents were asked to define the term “over-surveyed” via an open-ended question. 
Not surprisingly, the most frequent definition offered was: “Too many surveys.” However, the 
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overall content analysis suggests a more complex definition: the combination of frequent surveys 
that are perceived as irrelevant to daily student (cadet) life (“Too many surveys about nothing 
pertinent.”)- Furthermore, many respondents were very clear about the impact of frequent 
surveys perceived as irrelevant by students: “Too many surveys are given and this number 
causes survey takers to care less and less about their response because of the monotony.” 
Respondents described a combination of other factors that impacted their definition of the term 
“over-surveyed”: perceived lack of stated purpose, failing to provide feedback to participants, 
too many surveys that focus on every minute aspect of the students’ lives, and reduced personal 
time already strained by rigorous academy and military training requirements. 

Students’ View of the Optimal Survey 

Given these factors, the implementation of measures designed to ameliorate the impact of 
feeling “over-surveyed” becomes critical. Depending upon the circumstances, reducing the 
response burden could potentially result in higher response rates and increased data validity. 
Previous analyses, as well as the results of the current survey, highlighted the validity issues that 
arise when students bubble in a tree on the answer sheet (“christmas-treeing”) or mark the same 
option regardless of the question. According to the respondents, on average , a survey should 
consist of 22 items and require 13 minutes or less to complete. Further, respondents indicated 
that surveys should be administered only three to four times per school year. For the current 



survey, the overall completion time was 12.5 minutes. 



Make it simple and quick and people 



will respond and respond honestly.” 

When asked about when they prefer to complete surveys, 83% preferred a designated 
time rather than their personal time (6%). “We have limited personal time as it is; when the time 
is set aside, it is easier to complete it.” “If I have a time set aside, I cannot put it off and then 
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eventually decide to not do it.” Eleven percent did not indicate a preference. Freshmen were 
more likely to request a designated time (91%). Juniors and seniors were more likely to choose 
personal time (10% and 12% respectively). 

Student Confidentiality Concerns and Electronic Surveys 

More than one-third of the respondents indicated that they were “concerned” (21%) or 
“somewhat concerned” (18%) about the confidentiality of their electronic survey responses. 
Seniors (concerned = 27%; somewhat concerned = 21%) and women (concerned = 22%; 
somewhat concerned = 26%) were more likely to express confidentiality concerns. Four percent 
of all respondents indicated that they knew of an instance in which students were identified based 
on their survey responses. Both seniors (11%) and women (10%) were more likely to report 
knowledge of such an incident. 

USAFA surveys are increasingly electronic in format. When asked about suggestions for 
effectively communicating the safeguards used to ensure respondent confidentiality, the most 
frequent input was “Tell us.” Interestingly, these safeguards are described in nearly every 
USAFA survey, so these comments suggest that some students may not thoroughly review survey 
instructions. A number of students did recognize that they had already been informed of the 
safeguards in the current survey. Other suggestions included: demonstrating the efficacy of the 
survey software’s data encryption algorithm, sending an e-mail to all surveys describing the 
safeguards, avoid asking for a student’s username and password, explaining the confidentiality 
enhancements at the beginning of the survey, and placing a description of the enhancements on a 
separate screen before the survey begins. 

Student Motivation and Survey Participation 

When asked about their perspective on survey participation as an Air Force member, 4 1 % 
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of the respondents indicated that they felt a sense of duty toward providing data to their 
leadership. Conversely, 23% indicated that they were not interested in completing ANY survey. 
Women (50%) and freshmen (49%) were more likely to feel a sense of duty. Sophomores (32%) 
were least likely to feel a sense of duty in relation to survey participation. Reflecting a moderate 
perspective, 32% reported that they understood the need for survey data, but that they were not 
interested in providing such data. Given the hectic pace of student life at USAFA, it is not 
surprising that 47% of the respondents indicated that they view surveys as “one more thing that I 
have to do.” When comparing seniors to freshman, this perspective is even more pronounced. 
Seniors (52%) endorsed this option more often when than freshmen (42%). 5 However, while 
surveys are typically viewed as one more request for student time, respondents indicated that they 
understood the value and purpose of surveys. “I feel they are necessary, but at the same time, I 
feel so loaded down at times. Surveys take too much valuable time.” 

When asked if cadets should receive privileges for their survey participation, 41% 
responded “Yes, more cadets would respond to surveys” while 54% said “No” and 5% described 
an “Other” alternative. Seniors (22%) were least likely to agree with the use of privileges. “If 
cadets were offered privileges (e.g., sleeping in, weekend passes), the survey data would 
subsequently become less accurate because people would hurry through in order to receive the 
privilege.” Conversely, sophomores (55%) were most likely to endorse the use of privileges to 
encourage participation. Privileges most often suggested were sleeping in and disbursing more 
weekend and day passes for off-base activities. “I think that privileges would encourage cadets 
to take them more seriously.” 



Respondents could select more than one answer for this question; hence, the total exceeds 100%. 
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Finally, 25% of the respondents indicated a willingness to serve as a student survey 
reviewer prior to the administration of a given survey. Another 42% indicated a willingness but 
felt too busy to serve in this role. Overall, over two-thirds (67%) expressed an interest toward 
involvement in survey development. 

Discussion 

The survey data provides clear empirical support for the anecdotal data discussed 
previously. A large majority of the respondents felt over-surveyed and many respondents clearly 
noted the relationship between “too many surveys” and the resulting negative impact on the 
validity of survey results. Also, the definition of the term “over-surveyed” was not as 
straightforward as one might suspect. The frequency of survey administrations during an 
academic year is a major contributor. Further, the qualitative data suggests that other factors 
interact with frequency to varying degrees. These factors include the time required to complete 
surveys, survey length, a clear statement of the survey’s purpose, lack of timely feedback to 
respondents, and lack of perceived relevance to the students’ lives. 

On an encouraging note, at least half of the respondents reported a moderate position 
(ranging from “Yes, but I realize that the data are valuable” and “Somewhat” to “Somewhat, but 
I realize the survey data are valuable.”). Even so, the first challenge is to convince the remaining 
two-thirds that survey data are an important part of making effective decisions that can 
significantly impact their lives. The survey research results reported here provide important 
guidelines that should be considered in student surveys in terms of survey length, duration, and 
frequency of administration. 

Respondents also provided clear guidance for future USAFA surveys. During the last 2.5 
years, analysis of electronic survey completion times consistently demonstrated that the average 
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completion time for electronic USAFA surveys ranges from 12 to 13 minutes. The current 
survey serves as a perfect example: the average completion time was 12.5 minutes. Furthermore, 
on average, respondents indicated that the optimal completion time for any survey is 13 minutes. 
In fact, these analyses suggest that developing a USAFA survey that requires more than 13 
minutes may be a case of diminishing returns. Based upon current and previous analyses, 
students appear to place a limit on the time that they are willing to invest. In terms of frequency 
of administration, the results indicate that the use of sampling should continue and that the 
number of survey administration should be limited. Perhaps one solution is to restrict survey 
administrations to specific times in a given year, thus mitigating the perception that too many 
surveys are administered at USAFA. 

Clearly, there are some caveats to consider during survey design and administration. The 
number of questions is easily confounded by the type of question. Forced-choice questions 
require less time to answer when compared to a well-developed response to an open-ended 
question. Respondents can quickly respond to five to six Likert-scaled items per minute. 
However, open-ended question requires significantly more time (depending upon factors such as 
complexity of the question, the respondents knowledge and experience with the topic). Pilot 
testing, knowledge of the student population, and the purpose of the survey should be used to 
ensure that instruments generally fall within the student-identified parameters described above. 

If a survey reflects complex constructs such as gender attitudes, social support, and effective 
leadership, then the students’ view of the “optimal” survey typically becomes more difficult to 
apply. 

In terms of the electronic surveys, the results indicate that USAFA is making significant 
progress toward gaining student trust in the area of data confidentiality. However, significant 
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and sustained effort is necessary. The fact that seniors tend to distrust the assurance of 
confidentiality as compared to freshmen indicates that breaches do occur and that the impact is 
probably cumulative. Hence , efforts to secure and maintain the trust of incoming freshman 
should be sustained and perhaps intensified through the four years at the Academy. Significant 
levels of trust can be lost with even a small slip in confidentiality, or more likely, the unfounded 
rumor that such a breach has occurred. 

Regarding the students’ motivation underlying survey participation — feeling a “sense of 
duty” or viewing surveys as “just one more thing that I have to do” appears to be the driving 
force behind the relatively successful response rates observed at USAFA. However, the goal of 
enhancing the validity of survey data will only be met through higher response rates, and 
necessarily, through providing an effective case to students regard the need for and use of survey 
data. An effective case is based upon, among other things, a clear purpose, effective 
administration procedures, and timely feedback of the respondents’ impact upon decisions that 
clearly impact students’ daily life. 

Future research should be conducted to further elucidate the relationship between 
frequency of survey administration and the factors that combine to impact feeling “over- 
surveyed,” e.g., in what ways does low perceived relevance increase student frustration? To 
what degree does this frustration impact data validity via decreased participation and reliability? 
Furthermore, the data indicate that a core set of students may exist who are extremely difficult to 
engage in the survey research process. This group should be studied more closely in order to 
better understand their perspectives. In addition, a decision to provide a reward for participation 
should receive careful study. While many of the younger students eagerly supported this option, 
the older students clearly described the impact of incentives on data validity. 
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